Apparatus and method for constructing a virtual 3D model from a 2D ultrasound video

ABSTRACT

A method for creating a three-dimensional image of an object from a two-dimensional ultrasound video is provided. The method includes acquiring a plurality of two-dimensional ultrasound images of the object and recording a plurality of videos based on the acquired two-dimensional ultrasound images. Each of the videos includes a plurality of frames. The method further includes separating each of the plurality of frames, cropping each of the plurality of frames to isolate structures intended to be reconstructed, selecting a frame near a center of the object and rotating the image to create a main horizontal landmark, and aligning each frame to the main horizontal landmark. The method also includes removing inter-frame jitter by aligning each of the plurality of frames relative to a previous frame of the plurality of frames, reducing the noise of each of the frames, and stacking each of the frames into a three-dimensional volume.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/401,952 entitled “APPARATUS AND METHOD FORCONSTRUCTING A VIRTUAL 3D MODEL FROM A 2D ULTRASOUND VIDEO” which wasfiled Sep. 30, 2016. The entirety of the aforementioned application isherein incorporated by reference.

BACKGROUND

Ultrasonic imaging is a popular technique in medicine used to createvisual representations of the interior of a body for clinical analysisand medical intervention. However, the scan provides eithertwo-dimensional images or a video and no easy way to filter out noise,which confines the accuracy of the analysis.

Ultrasonography is used routinely in ophthalmic examination to evaluatestructures in the eye, to rule out pathology and follow certainpathologic and physiologic changes in the eye. The scan, however,provides a two-dimensional image which somewhat limits the detailedevaluation of the tissues. By converting the two-dimensional media intoa three-dimensional model, it is easier to see the contours and finedetails of the tissue without having to mentally approximate frame byframe. This allows the user to understand, and helps recognizing, subtleproblems which would be otherwise missed.

Currently, few methods and apparatuses achieve such three-dimensionalmodels and are rarely used in the ophthalmology field.

SUMMARY

The following presents a simplified summary of the invention in order toprovide a basic understanding of some example aspects of the invention.This summary is not an extensive overview of the invention. Moreover,this summary is not intended to identify critical elements of theinvention or to delineate the scope of the invention. The sole purposeof the summary is to present some concepts in a simplified form as aprelude to the more detailed description that is presented later.

Provided are a plurality of example embodiments, including, but notlimited to methods and devices for creating a three-dimensional image ofa body part, such as the eye, for example, in order to diagnose problemsand determine solutions.

In one general aspect, a method for creating a three-dimensional imageof an object from a two-dimensional ultrasound video is provided. Themethod includes acquiring a plurality of two-dimensional ultrasoundimages of the object and recording a plurality of videos based on theacquired two-dimensional ultrasound images. Each of the plurality ofvideos comprises a plurality of frames. The method further includesseparating each of the plurality of frames and cropping each of theplurality of frames to isolate structures intended to be reconstructed.The method also includes selecting a frame near a center of the objectand rotating the image to create a main horizontal landmark. The methodfurther includes aligning each of the plurality of frames to the mainhorizontal landmark. The method also includes stacking each of thealigned plurality of frames into a three-dimensional volume.

In another general aspect, the method for creating a three-dimensionalimage of an object from a two-dimensional ultrasound video includesremoving inter-frame jitter by aligning each of the plurality of framesrelative to a previous frame of the plurality of frames.

In another general aspect, the method for creating a three-dimensionalimage of an object from a two-dimensional ultrasound video includesreducing a noise of each of the plurality of frames.

In another general aspect, the object is a part of a human body.

In another general aspect, the object is a human eye.

In another general aspect, the plurality of two-dimensional ultrasoundimages are B-scan images of the human eye.

In another general aspect, the acquiring a plurality of two-dimensionalultrasound images is performed by a Quantel ultrasonic biomicroscope ora B-scan probe.

In another general aspect, each of the plurality of frames represents anindividual slice of a particular segment of one of the two-dimensionalultrasound images in one dimension.

In another general aspect, the cropping each of the plurality of framescomprises excluding extraneous parts of the images.

In another general aspect, the selecting the frame near the center ofthe object and rotating the image rotates all other frames by the sameamount.

In another general aspect, the reducing a noise of each of the pluralityof frames comprises low-pass filtering or median filtering.

In another general aspect, the stacking each of the plurality of frameswith reduced noise into a three-dimensional volume comprises removinglayers of the three-dimensional image.

In another general aspect, the removing of layers of thethree-dimensional image is performed by using planes to exclude voxelsoutside of a cuboidal region specified by a user.

In one general aspect, a method for creating a three-dimensional imageof an object from a two-dimensional ultrasound video is provided. Themethod includes obtaining a plurality of two-dimensional ultrasoundvideo images of the object. Each of the plurality of two-dimensionalultrasound video images comprises a plurality of individual frames. Themethod further includes isolating each of the plurality of individualframes from each of the plurality of two-dimensional ultrasound videoimages. The method also includes individually processing each of theplurality of individual frames to improve their suitability forconversion. The method further includes assembling together theindividually processed plurality of individual frames to create thethree-dimensional image of the object.

In another general aspect, the assembling together the individuallyprocessed plurality of individual frames uses landmarks that identifycommon areas in the individually processed plurality of individualframes.

In another general aspect, the assembling together the individuallyprocessed plurality of individual frames aligns each of the individuallyprocessed plurality of individual frames based on image boundaries inadjacent frames of the individually processed plurality of individualframes.

In another general aspect, the individually processing each of theplurality of individual frames comprises selecting an individual framenear a center of the object and rotating the individual frame to createa main horizontal landmark.

In another general aspect, the assembling together the individuallyprocessed plurality of individual frames aligns each of the individuallyprocessed plurality of individual frames based on the main horizontallandmark.

In another general aspect, the assembling together the individuallyprocessed plurality of individual frames comprises creating an objectthat contains a three-dimensional array of values corresponding toslices, rows, and columns, wherein the slices are enumerated from top tobottom, the rows from back to front, and the columns from left to right.The assembling together the individually processed plurality ofindividual frames further comprises visualizing the three-dimensionalimage of the object by using volume rendering that displays a projectionof discretely sampled three-dimensional data values, wherein each volumeelement is represented by a single value or a list of values. Theassembling together the individually processed plurality of individualframes also comprises outputting a three-dimensional image format as apicture of a projection of the image.

Other features and aspects may be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the example embodiments described hereinwill become apparent to those skilled in the art to which thisdisclosure relates upon reading the following description, withreference to the accompanying drawings, in which:

FIG. 1 is a schematic view of individual frames, which representindividual slices in one dimension of two dimensional video images;

FIG. 2 is a schematic illustration of the selection of a frame among theindividual frames near the center of the eye and the rotation of theimage;

FIG. 3 is a schematic illustration of the registration of the individualframes relative to one another;

FIG. 4 is an image of a 3D reconstructed resulting model of an eye;

FIG. 5 is a schematic illustration of an example apparatus forcollecting raw frames in a controlled and consistent fashion;

FIG. 6 is a perspective view of an example adapter which attaches thetranslation stage to the surgical microscope; and

FIG. 7 is a perspective view of an example probe holder.

Throughout the drawings and the detailed description, unless otherwisedescribed, the same drawing reference numerals will be understood torefer to the same elements, features, and structures. The relative sizeand depiction of these elements may be exaggerated for clarity,illustration, and convenience.

The figures show various aspects of embodiments of the invention, asdescribed in more detail hereinbelow.

DETAILED DESCRIPTION

Example embodiments that incorporate one or more aspects of theapparatus and methodology are described and illustrated in the drawings.These illustrated examples are not intended to be a limitation on thepresent disclosure. For example, one or more aspects of the disclosedembodiments can be utilized in other embodiments and even other types ofdevices. Moreover, certain terminology is used herein for convenienceonly and is not to be taken as a limitation.

The method described herein relates to conversion of two dimensional(2D) video images into a virtual three-dimensional (3D) object which canbe analyzed and manipulated. The method described herein utilizesultrasound videos of the eye. However, even though the primary focus ofthis description uses ultrasounds of the eye, it should be applicable toall ultrasound images of different body structures.

The described method involves reconstructing images of multiple slicesof the eye, similar to a CT scan three-dimensional (3D) reconstruction,to create an improved 3D image that provides a much needed overview ofthe ocular structures. The 3D image can be helpful in identifyinganatomical structures quickly and diagnosing pathologies with greateraccuracy.

3D reconstruction of the full anterior chamber can be used for automaticmeasurement of clinically relevant parameters. One example is theiridocorneal angle which has implications for aqueous outflow anddiseases such as glaucoma. 3D imaging allows 360-degree measurement ofthe angle at one time, in contrast to most other techniques whichmeasure the angle in a single plane or a fixed number of locations.Furthermore, measurement of the angle from a 3D volume is less prone toerrors caused by oblique sampling planes.

3D reconstruction also allows measuring the anterior chamber (AC) volumeand height. The AC volume is usually measured by a geometricapproximation which does not take into account the proper curvature ofthe iris or the variation around the optic axis. 3D reconstructionenables automatic segmentation of the entire AC with the associatedmeasurements at voxel-level accuracy.

Generally, the method utilizes a plurality of two dimensional ultrasoundimages (i.e., B-scan images) taken of a human organ, such as the eye, ina traditional or improved manner. With respect to eye images, theseimages are typically performed in a linear slicing mode which shows onlya single particular segment of the eye. First, the 2D image slices areprocessed in order to improve their suitability for conversion.

After that, the processed images are “assembled” together to create a 3Dimage from the original 2D images to better represent the actualphysical construction of the organ in order to show various structures,including damage, defects, or other items of interest, which can greatlyimprove ultrasonography in ophthalmology. All image analyses andmanipulations can be done using a system such as the Matlab® commercialpackage, Mathematica, ImageJ, some other commercial package, or usingcustomized software.

The image assembly process is aided by information provided in theimages themselves, such as “landmarks” that identify common areas in thedifferent images, or by using image boundaries in adjacent images to aidin such reconstruction of the actual organ in 3D from the series of 2Dimages. Commercially available software, open source software, and/orcustomized software can be used to implement the process, which couldalso utilize various manual steps as alternatives as well. However, atleast one or more of the steps of the method cannot be performedmanually. In addition, the creation of the 3D images permits obtainingvolumetric measurements and calculations about the imaged organ(s).

Furthermore, the process is improved by using a new machine to helpautomate the scanning process using various commercial probes, asopposed to the current manual process, to ensure consistent images andaid in the reconstruction process.

Example Methodology

In an example method, the images of the desired body part (e.g., humaneye) can be acquired with Quantel ultrasonic biomicroscope and withB-scan probe. A video of the acquired images may be recorded by themachine and can be translated into an Audio Video Interleaved (“AVI”)format. The videos in an example embodiment can be 10 seconds long, andmay include multiple individual frames of two dimensional imaging whilethe eye is scanned using an ultrasonic probe.

The videos are read in a movie, or a series of movies, and the frames,which represent individual slices in one dimension, are separated orisolated (as illustrated in FIG. 1).

Each frame is then cropped crudely, using video editing software, suchas Pinnacle Studio 18 or ImageMagick, for example, to show just the areaof interest, isolate the structures intended to be reconstructed, and toremove extraneous text and parts of the images, such as the blue frameand the patient name which are added by the manufacturer software.

Once the preparatory work for the video is done, for one exampleembodiment it is imported into a program encoded in the Matlab®commercial package, where it is then saved as a plurality of individualimages.

Next, the individual images are processed prior to their conversion intoa 3D image. Specifically, the user selects an individual frame near thecenter of the eye and rotates the image to make the iris (e.g., the mainhorizontal landmark) truly horizontal. This manipulation rotates allother individual frames by the same amount.

Next, the individual frames are registered relative to one another toremove inter-frame jitter. In laboratory conditions, jitter is basicallyabsent. However, when moving the probe by hand or when the probe is usedon a breathing patient, there is undesirable motion between the probeand the patient's eye. This results in jumps or waves in the 3Dreconstruction. As show in FIG. 2, to remove these jumps or waves, themethod aligns each individual frame to the one before it. Mathematicallyspeaking, as show in FIG. 3, the transformation (which is usuallyrestricted to translation or rotation) minimizes the numericaldifference between pixels in adjacent individual frames.

As a next step, the images are de-noised. Specifically, the conditionsunder which the original individual frames were recorded can be noisy.There are hundreds of algorithms (or filters) that can be used to reducethis noise. Some of the more effective algorithms have been low-passfiltering and median filtering, for example.

After the 2D images are de-noised, they are stacked into a 3D volumethat can be manipulated within custom software. For example, since theuser typically desires to see structures inside the eye, it may beimportant to have the ability to remove layers of the 3D images. Thisremoval is accomplished by simply using planes to exclude voxels outsideof a cuboidal region that the user can specify.

The conversion into 3D images is done by creating an object thatcontains a three-dimensional array of values (or lists of values) thatrepresent a 3D raster image. The first three dimensions of the objectcorrespond to slices, rows, and columns, respectively. Slices areenumerated from top to bottom, rows from back to front, and columns fromleft to right. Then a 3D object is visualized using a set of techniquescollectively known as volume rendering that display a projection ofdiscretely sampled 3D data values. Given an object containing a regularvolumetric grid, each volume element is known as a voxel and isrepresented by a single value (or list of values). Upon output, a 3Dimage formats as a picture of a projection of the image (not as a 3Darray of values). As shown in FIG. 4, a 3D model of an eye wassuccessfully created from an ultrasound video of that eye. This is asignificant achievement in a revolutionary diagnostic tool.

Example Apparatus and Control Software

A typical apparatus for obtaining a Cineloop video imaging of a B-scanultrasonography generally includes an ultrasonic probe attached to amotor system. The motor system can help in arranging the speed and thedirection of the Cineloop video imaging. However, an apparatus forobtaining a good quality Cineloop that also obtains images steadily andconsistently at a set speed is a challenge and is difficult to achieveeven by an experienced ultrasonographer or a surgeon with steady hands.Especially for a 3D reconstruction of a B-scan Cineloop, it is desirableto obtain a steady, reproducible, and reliable Cineloop.

Hence, an apparatus that could automatically scan the eye (or otherorgan) in a steady and consistent manner is desired using one or moredifferent types of ultrasonic probes, which will then provide improvedimages and hence provide a better final result. This device would ensurethat the ultrasonic imaging probe is moved at a constant, regular rate,and may operate with a number of different types of probes as desired tobroaden its applicability.

A secondary inaccuracy for the 3D model deals with alignment. Thevolumetric rendering algorithm naturally accounts for smallmisalignments when smoothing out frame to frame. However, if faced witha large wobble between pictures, it will still smooth out the frame,unknowing of the distorted volume it will create. Therefore, each imageneeds to be aligned to a landmark before it is made into a 3D model, orone image frame can be aligned to the adjacent image frame. Again, forthe purposes of this example, any images that were irreconcilably skewedmight be discarded. However, this can be improved by providing analignment algorithm that will set all images in a certain landmark. Aspecialized machine can also help with this process.

For the example approach, the entire process from importing the originalvideo, process all the images, and then creating the 3D model,advantageously takes only about 10-15 minutes. This varies depending onhow many images need to be thrown out. This can be improved by using thespecialized imaging machine, and additional improvements such as usingthe alignment algorithm.

The example apparatus 400, schematically illustrated in FIG. 5 anddescribed herein, will make obtaining such images a substantially easierprocess. This in turn will revolutionize performing and interpretingB-scan ultrasonography in ophthalmology and will eliminate the need fora very experienced ultrasonographer to be present to achieve therequired and aspired best images in ocular ultrasonography.

The example apparatus 400 includes an ultrasound probe 402, a motor 403connected to a motor controller 404 that powers and controls themovement of the probe, and sends the obtained digitized images to theconnected computer 401 for further processing.

Image acquisition may be performed by an Aviso™ 50 MHz UltrasoundBiomicroscopy (UBM) probe, such as UBM probes from Quantel Medical, forexample. The probe 402 rasters across the field of view capturingapproximately ten frames per second.

The probe 402 may be attached to an Aviso™ module which powers andcontrols the probe 402, and sends the digitized information to aconnected computer 401. The computer 401 stores the image sequencesdigitally in a proprietary Cineloop format, and can export them as AVIor JPEG formats.

Communication with the probe 402 uses the probe manufacturer'sproprietary software, whereas customized software or open sourcesoftware is utilized to control and/or communicate with the scanningmachine. Commercially available software, customized software, and/oropen source software can be utilized to process the resulting images,such as to practice a process such as described above.

The probe may be attached to a 2″ (or about 5 cm) linear translationstage, such as stage model MTS50-Z8 provided from Thor labs, forexample. The stage can achieve a 1.6-micron positional repeatability and±0.25 mm/sec velocity stability. The stage is mechanically attached to asurgical microscope, such as a Leica surgical microscope, for example,which provides stability as well as a foot-controlled mechanicalpositioning.

In one embodiment, the probe can be attached to the motor with anadapter (shown in FIG. 6), which is permanently fixed to the threadedholes on the translation stage and attaches the translation stage to thesurgical microscope. It utilizes the threaded holes on the underside ofthe microscope head which are provided for attachingmanufacturer-provided accessories. The adapter grips the probe like avice and is tightened with two thumb screws. The motor may be attachedto the microscope via a second adapter which utilizes the threaded holeson the fixed part of the translation stage and the adapter holes on thebottom of the microscope. In one embodiment, the adapter may be a 3Dprinted adapter.

As further illustrated in FIG. 7, a probe holder can attach to themobile part of the translation stage and can grip the probe handle,holding it parallel to the microscope's optical axis. The combination ofthe two adapters shown in FIG. 6 and FIG. 7 allows the probe to bepositioned using the microscope's own controls. In one embodiment, theprobe holder may be a 3D printed probe holder.

The motor can be controlled via a Thorlabs DC servo controller which canattach to the computer via a USB port. The controller allows either acrude manual positioning of the motor via a toggle switch or a precisecontrol through the computer interface.

For automatic operation, the probe is turned on and off via a Numatosingle-channel USB relay 408 connected to the foot pedal port on theAviso™ module. The relay 408 is wired such that the pedal remainsoperational.

Both the relay 408 and the motor controller 404 are connected to thesame computer 401 running the Aviso™ software 407. The computer 401 runsa second, custom program 406 written in Python which coordinates theprobe 402 motion with the probe data acquisition. This second customprogram 406 allows the user to specify a region of the eye for dataacquisition and the number of frames to be captured.

In manual operation, the probe is typically designed to be turned on andoff by a foot pedal. The software provided by the manufacturer allowsthe user to capture a sequence of 2D images in the form of a multi-framemovie.

In one embodiment, the number of frames may be limited to cap theresolution in the swept direction to a certain number of pixels. In thisembodiment, the approach would be to take several movies over smallerregions of the eye and combining the movies once the data acquisition iscompleted. Therefore, the start of a movie must be synchronized with themotion of the stage which can be accomplished by replacing the footpedal with a computer-controlled relay from Numato lab. Wiring to theprobe interface required the construction of a simple custom cable.

The stage is controlled by software prepared using Python. In oneembodiment, the method may use as an input the starting and endingposition of the stage and the number of segments into which that rangeshould be divided. The method then may calculate the starting andstopping position of each division, add a little margin for stageacceleration, and ultimately compute the speed at which the stage mustmove to cover each division in exactly ten seconds (or whatever amountof time corresponds to one video recorded by the probe software). As themethod may start and stop the stage on each pass, it may turn the probeon and off by toggling the relay.

In one embodiment, the ultrasonic probe may be seated on an immersioncup and may be thereby attached to the motor system.

In an embodiment, 3D reconstruction may be facilitated by intraocularcontrast agents, such as lipid-bound fluorocarbon micro- andnano-bubbles. These agents may be injected into the anterior chamber tohighlight structures, such as, but not limited to, the AC volume itselfand the aqueous outflow pathway, including Schlemm's canal. The agentsmay also be used for making dynamic measurements, such as the aqueousoutflow facility.

Alternative Image Reconstruction Methodology

As an additional approach, after the ultrasound video is isolated intoindividual frames, using basic image analysis techniques, to each frameis applied a filter, such as a Sobel filter, a Canny filter, a binaryfilter, or another process, to outline the edges of the structure. Next,the object outlined by the filter is filled in to create a binarygradient image. Finally, each frame is stacked together, given a depth,and is applied contour analysis to smooth out the overallthree-dimensional object.

This approach can utilize different methods of processing the 2D imagesbefore reconstruction, such as image segmentation and manualbinarization, or some other process. The processes are used to achievethe goal of creating a binary image. Different methods can be used fortraining and familiarity with the concepts, and to provide alternativesolutions. For image segmentation, a filter is applied to each frame ofthe video that will allow the computer to sense the edges of the objectof interest. Then the edges are filled in using cell detectionalgorithms to create a solid blob that outlines the object. For manualbinarization, a histogram of the colors is created for a random image.Then, the low and high thresholds are determined through sliders. Valuesoutside of the thresholds are black, the ones inside are white. The sametwo thresholds can be used for all consequent images. Finally, all ofthe images are stacked together and form a three dimensional model. Thiscan be done using a commercial package such as Mathematica, ImageJ, orthe Matlab® commercial package, or using customized software, forexample. This program can be provided in the mobile realm using acustomized application, such as using coding as a JavaScript package ora more complex and complete application.

The program processes the images in two different ways, both aiming forthe same result. Image segmentation outlines the structure of thetissues using image segmentation algorithms as seen in the Matlab®commercial package online toolbox. Image segmentation for this exampleapproach runs in six steps: importing the images, applying filters foredge detection, dilating the image, filling the interior gaps, removingconnected objects on border, and finally smooth out the object.

After image segmentation is complete, contour analysis is applied asprescribed in the Matlab® commercial package toolbox to align each imagealong a landmark (e.g., the bottom of the iris) and then stack themtogether to reconstruct a three dimensional model. Details outlining thespecifics of each step will appear in the following paragraphs. Forillustrative and procedural purposes, the figures provided for theexample process are all of cells and not of an eye, but the basictechnique remains the same.

Importing the Images

The ultrasound video can include an arbitrary number of frames, whichcan be imported as a whole into a temporary working folder. The videomay then be converted into a number of individual images correspondingto the number of frames, and the individual images can be saved in thesame folder using the VideoReader and imwrite functions provided in theMatlab® commercial package, for example.

In one embodiment, for the example, the ultrasound video may include 100frames and the video can be converted into 100 individual images andsaved in the same folder using the VideoReader and imwrite functionsprovided in the Matlab® commercial package, for example.

Applying Sobel and Canny Filters

Sobel and Canny are both edge detection filters that utilize 2D spatialgradient measurements to create a binary image. Edge detection refers tothe process of identifying and locating sharp discontinuities in animage. The discontinuities are abrupt changes in pixel intensity whichcharacterize boundaries of objects in a scene. Edges in images are areaswith strong intensity contrasts. Edge detecting an image significantlyreduces the amount of data and filters out useless information, whilepreserving the important structural properties in an image. While theMatlab® commercial package has both filters as a built in function, forthe purposes of example custom functions have been provided.

Dilating the Image

The binary gradient image shows lines of high contrast that indicate theedges of the object. These lines do not quite delineate the outline ofthe object of interest and gaps can be seen in the lines surrounding theobject in the gradient mask. These linear gaps will disappear if theSobel filter image is dilated using linear structuring elements, whichcan be created with the strel function. The binary gradient mask isdilated using the vertical structuring element followed by thehorizontal structuring element.

Filling the Interior Gaps

The dilated gradient mask can show the outline of the objectappropriately, but there may still be holes in the interior of theobject. To fill these holes, the imfill function may be used.

Removing Connected Objects on the Border

The object of interest has been successfully segmented, but it is notthe only object that has been found. Any objects that are connected tothe border of the image can be removed using the imclearborder function.The connectivity in the imclearborder function was set to 4 to removediagonal connections.

Smooth Out the Object

Furthermore, in order to make the segmented object look natural, theobject is smoothed by eroding the image twice with a diamond structuringelement. The diamond structuring element is created using the strel orbwperim function. The final product is a noise free binary image.

For manual binarization there are a total of 5 steps; conversion tograyscale, selecting a representative picture, creating a histogram ofthe values of each shade, finding the minimum and maximum thresholds,and finally creating the binary image. Note that although the manualprocess is described for the purpose of highlighting the process, suchprocedures are automated in the final procedure.

Converting to Grayscale

The function which converts the ultrasound video to individual framessaves each image as a RBG, rather than a grayscale. These images areprocessed as a grayscale image because of the interest in creating abinary image.

Selecting a Representative Image

For the example approach only about half of the images from theultrasound may be used. Since the scan takes a period of 10 seconds tocomplete, usually more than one scan of different parts of the eye isperformed in multiple segments, and more than one image is obtained. Anyincomplete images are automatically thrown out. Additionally, thrown outare any images that may have been corrupted due to errors intransferring data. From the collection of images that remain, the firstone is selected to determine the thresholds.

Creating a Historgram

A grayscale image is represented with number ranging from 0 to 100, with0 being pure black and 100 being pure white. By creating a histogram ofthe values for each individual pixel, it can be seen which values makeup the area of interest. The histogram is made with the function imhist.The histogram will then be used in the following step.

Finding Thresholds

Here is manually selected the thresholds for a binary filter. Thesethresholds will be applied to all the other images in the collection.The program will prompt an interactive GUI with the original monochromeimage, the masked image outside the thresholds, and the final binarizedimage. There are interactive sliders that correspond to values on thehistogram where the user can manually change the thresholds in real timeto see the final binary image.

Creating the Binary Image

The thresholds represent the window of values of interest. The imagesare taken as an array of their individual grayscale values. Everythingwithin the thresholds is changed to 1, and everything outside thosevalues is changed to 0. The end result is the original picture beingconverted into a binary one. As an example, after the interactive GUI iscomplete, it will bring up a number of masked images corresponding todifferent imaging analysis techniques. These may not be used later on inthe project, but may be useful for training purposes.

The final step is the conversion into 3D. As described above, this isdone by creating an object that contains a three-dimensional array ofvalues (or lists of values) that represent a 3D raster image. The firstthree dimensions of the object correspond to slices, rows, and columns,respectively. Slices are enumerated from top to bottom, rows from backto front, and columns from left to right. Then a 3D object is visualizedusing a set of techniques collectively known as volume rendering thatdisplay a projection of discretely sampled 3D data values. Given anobject containing a regular volumetric grid, each volume element isknown as a voxel and is represented by a single value (or list ofvalues). Upon output, a 3D image formats as a picture of a projection ofthe image (not as a 3D array of values).

Results

As mentioned previously, one goal of this approach is to provide basicimage enhancement techniques and image analysis techniques to producethe 2D to 3D ultrasound converter. All the steps listed in themethodology have been provided in a working example.

That being said, while the code functions correctly, the Sobel and Cannyfilters do not optimally outline the edges of the eye due to lack ofcontrast between the bright and dark parts. This is a problem with themain source of obtaining the video. The ultrasound does produce highercontrast videos, but at the cost of a drastic increase in backgroundnoise. The contours of interest, the iris, cornea, and sclera, appeartoo grainy and grey from the ultrasound video to properly outline usingregular 2D spatial gradient algorithm.

The complete binarized image is better provided with better filters.However, with regards to the image segmentation analysis, each step isin working order.

The last and most important step is the actual creation of the 3D model.Simply put, each binary image is stacked in a 3D array along thez-direction. The images are then aligned and stabilized along the lensof the eye. Then through volume rendering algorithms, the gap betweeneach corresponding white tile is smoothened while all black tiles areleft as a blank space. The algorithm runs in all 3 planes to correct anyabnormalities that occur if done in only one direction. Both thealignment and volumetric rendering algorithms are predeterminedfunctions available through Mathematica. The resulting volume can bemanipulated in the 3D space and the entire object can be rotated 360degrees in any direction. Any ridges and deformation can be clearly seenin a 3D setting, compared to the original 2D images.

Discussion

The overall results of the method are in line with the main goal of theproject. A 3D model of an eye was successfully created from anultrasound video of that eye. This is a significant achievement in arevolutionary diagnostic tool. However, there are a number ofimprovements to be provided in additional examples. Firstly, theapplication of the Canny and Sobel filters can be improved upon. Afundamental problem with the way these filters worked impeded progress.Recall that both Canny and Sobel filters are edge detecting filters.They operate by discerning color gradients in the image and then map outa gradient with a drastic change. None of the gradients were largeenough to detect major changes. The ultrasound produces a grayscaleimage that, by definition, is only shades of black and white. Without alarge RBG color gradient to discern edges, the filter becomes lessuseful. For this reason a manual binarization program to accuratelyproduce the binary images was used. Using a color image as a startingpoint could also improve the process.

Nevertheless, the resulting 3D model is workable, and the very firststep in developing a high resolution model. While it does portray theacute deformations of the eye, it is still inaccurate to a certaindegree. About half of images were thrown out before the final collectiondue to incomplete scans, distorted images, or images that were tooblurry to use. In order to increase the resolution of the 3D model,ideally all 100 frames of the ultrasound video would be of one singlepass through the eye, and all such frames are utilized. The process canbe improved by making sure that one pass lasts exactly 10 seconds, thelength of time it takes to film 100 frames. This is difficult, becauseit requires an extremely steady hand to move the ultrasound probe at aconstant speed just 1 inch in 10 seconds.

Many other example embodiments can be provided through variouscombinations of the above described features and specific embodiments.Although the embodiments described hereinabove use specific examples andalternatives, it will be understood by those skilled in the art thatvarious additional alternatives may be used and equivalents may besubstituted for elements and/or steps described herein, withoutnecessarily deviating from the intended scope of the application.Modifications may be necessary to adapt the embodiments to a particularsituation or to particular needs without departing from the intendedscope of the application. It is intended that the application not belimited to the particular example implementations and exampleembodiments described herein, but that the claims be given theirbroadest reasonable interpretation to cover all novel and non-obviousembodiments, literal or equivalent, disclosed or not, covered thereby.

What is claimed is:
 1. A method for creating a three-dimensional imageof an object from a two-dimensional ultrasound video, the methodcomprising: acquiring a plurality of two-dimensional ultrasound imagesof the object; recording a plurality of videos based on the acquiredtwo-dimensional ultrasound images, each of the plurality of videoscomprising a plurality of frames; separating each of the plurality offrames; cropping each of the plurality of frames to isolate structuresintended to be reconstructed; selecting a frame near a center of theobject and rotating the selected frame to create a main horizontallandmark; aligning each of the plurality of frames to the mainhorizontal landmark; and stacking each of the aligned plurality offrames into a three-dimensional volume.
 2. The method according to claim1, further comprising: removing inter-frame jitter by aligning each ofthe plurality of frames relative to a previous frame of the plurality offrames.
 3. The method according to claim 1, further comprising: reducinga noise of each of the plurality of frames.
 4. The method according toclaim 3, wherein the reducing the noise of each of the plurality offrames comprises low-pass filtering or median filtering.
 5. The methodaccording to claim 3, further comprising stacking each of the pluralityof frames with reduced noise into a reduced noise three-dimensionalvolume by removing layers of the three-dimensional image.
 6. The methodaccording to claim 5, wherein the removing layers of thethree-dimensional image is performed by using planes to exclude voxelsoutside of a cuboidal region specified by a user.
 7. The methodaccording to claim 1, wherein the object is a part of a human body. 8.The method according to claim 7, wherein the object is a human eye. 9.The method according to claim 8, wherein the plurality oftwo-dimensional ultrasound images are B-scan images of the human eye.10. The method according to claim 1, wherein the acquiring a pluralityof two-dimensional ultrasound images is performed by a Quantelultrasonic biomicroscope or a B-scan probe.
 11. The method according toclaim 1, wherein each of the plurality of frames represents anindividual slice of a particular segment of one of the two-dimensionalultrasound images in one dimension.
 12. The method according to claim 1,wherein the cropping each of the plurality of frames comprises excludingextraneous parts of the images.
 13. The method according to claim 1,wherein the selecting the frame near the center of the object androtating the image rotates all other frames by the same amount.
 14. Amethod for creating a three-dimensional image of an object from atwo-dimensional ultrasound video, the method comprising: obtaining aplurality of two-dimensional ultrasound video images of the object, eachof the plurality of two-dimensional ultrasound video images comprising aplurality of individual frames; isolating each of the plurality ofindividual frames from each of the plurality of two-dimensionalultrasound video images; individually processing each of the pluralityof individual frames to improve their suitability for conversion,wherein the individually processing each of the plurality of individualframes comprises selecting an individual frame near a center of theobject and rotating the individual frame to create a main horizontallandmark; and assembling together the individually processed pluralityof individual frames to create the three-dimensional image of theobject.
 15. The method according to claim 14, wherein the assemblingtogether the individually processed plurality of individual frames useslandmarks that identify common areas in the individually processedplurality of individual frames.
 16. The method according to claim 14,wherein the assembling together the individually processed plurality ofindividual frames aligns each of the individually processed plurality ofindividual frames based on image boundaries in adjacent frames of theindividually processed plurality of individual frames.
 17. The methodaccording to claim 14, wherein the assembling together the individuallyprocessed plurality of individual frames aligns each of the individuallyprocessed plurality of individual frames based on the main horizontallandmark.
 18. The method according to claim 14, wherein the assemblingtogether the individually processed plurality of individual framescomprises: creating an object that contains a three-dimensional array ofvalues, the three-dimensional array of values corresponding to slices,rows, and columns, wherein the slices are enumerated from top to bottom,the rows from back to front, and the columns from left to right;visualizing the three-dimensional image of the object by using volumerendering that displays a projection of discretely sampledthree-dimensional data values, wherein each volume element isrepresented by a single value or a list of values; and outputting athree-dimensional image format as a picture of a projection of theimage.